Skip to content

YOLO end-to-end example, CircularTensor hardening, CPU backend, ThreadFusion, multi-GPU routing - #2

Merged
morousg merged 3 commits into
Libraries-Openly-Fused:mainfrom
johnnynunez:post-merge-updates
Jun 11, 2026
Merged

morousg merged 3 commits into
Libraries-Openly-Fused:mainfrom
johnnynunez:post-merge-updates

Conversation

@johnnynunez

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #1 (thanks for the fast merge!) with the work that landed after the merge cut:

1. Real end-to-end example: YOLOv8 inference (examples/11_yolo_inference.py)

JPEG -> ONE fused GPU kernel (letterbox Resize(640,640, PRESERVE_AR, bg=114) + Div(255) + TensorSplit HWC->CHW) -> YOLOv8n ONNX -> NMS -> boxes. Verified on bus.jpg: 4 persons + 1 bus, correct boxes; fused preprocess ~0.55 ms steady-state. examples/fetch_yolo_assets.sh downloads the assets.

2. CircularTensor hardening (one real bug fixed)

Bug: generated ct_update/ct_snapshot baked W/H as C literals while the cache signature doesn't include size -> two same-signature CTs with different sizes cache-hit the wrong .so (OOB, verified). Fix: sizes live in the PyCT struct, read at runtime (the values-vs-types rule applied consistently). Plus: race audit (seq1's written temp slot proven disjoint from seq2's read slots for both orders × all rotation indices), temp/data coherence across 3 wraps, generate_source() API, and CI now compile-checks CircularTensor codegen too. 21 checks.

3. CPU backend (target="cpu")

Same fused chains through TransformDPP<ParArch::CPU> on host memory, numpy in/out, no CUDA required. Compiled with clang++ (note: g++ rejects Stream_<ParArch::CPU>'s ctor spelling — possibly worth an upstream look). CPU==GPU cross-validated.

4. ThreadFusion (thread_fusion=True)

Emits TransformDPP<GPU_NVIDIA, TF::ENABLED>. Auto-fallback to scalar when row bytes aren't 16-aligned (external tight-pitch pointers fault on vectorized loads — found the hard way with a prime-width image).

5. Multi-GPU device routing

device=N on DeviceBuffer/CircularTensor; pipelines follow the input's device; DLPack device id propagated. Honest caveat: developed on a 1-GPU box, exercised on device 0.

6. Torch integration suite (real torch 2.12+cu130)

Zero-copy both directions (mutation-visibility proven), external torch streams, out=, HWC->CHW vs torch reference, CircularTensor->torch, batch HF from a torch list. Found+fixed an _alloc_out shape bug for TensorSplit chains.

Also includes spikes/repro_divergent_sum_z.cu — the reproducer behind FusedKernelLibrary#250.

Verification

13 suites × 2 compilers (nvcc + clang) + cpu suite + torch suite, all green on RTX PRO 6000 (sm_120, CUDA 13.3) against LTS-C++17.

…ssing

examples/11_yolo_inference.py: JPEG -> ONE fused FKL kernel -> YOLOv8n.

The preprocessing that ultralytics does in several numpy/torch steps is a
single fused kernel: TensorRead (uchar3, any size) -> Resize(640x640,
bilinear, PRESERVE_AR letterbox with the YOLO gray-114 padding) ->
Div(255) -> TensorSplit (HWC -> planar CHW float, NCHW-ready). Standard
decode+NMS postprocessing in numpy; inference via onnxruntime.

Verified on bus.jpg (810x1080): 4 persons + 1 bus detected with correct
boxes mapped back through the letterbox transform. Steady-state fused
preprocess: ~0.55 ms (incl. D2H); kernel compiles once then disk-cached.

examples/fetch_yolo_assets.sh downloads bus.jpg + a COCO-pretrained
yolov8n.onnx; assets/ and .venv-yolo/ gitignored.
…ce tests

REAL BUG found while hardening: the generated ct_update/ct_snapshot baked
self.width/self.height as C literals, but the compile-cache signature only
contains dtype/batch/order/layout/chain — NOT the size. Two CircularTensors
with the same signature and different sizes would cache-hit the first .so
and run with the WRONG hardcoded grid and snapshot byte count (out-of-bounds
reads/writes; surfaced as 'invalid argument' in an unrelated test). Fix:
PyCT now stores w/h at ct_create time and the kernel launch + snapshot use
them at runtime, matching the values-vs-types rule everywhere else.
CODEGEN_VERSION 6->7.

New CircularTensor tests (21 checks total now):
- t_race_audit: proves the divergent update is race-free by construction --
  the temp slot written by seq1 is never among the slots read by seq2, for
  both orders and every rotation index.
- t_temp_data_consistency: temp/data double-buffer coherence across 3 full
  rotation wraps, verified plane-by-plane via ct_snapshot_temp.
- t_generate_source_no_compile: new generate_source() API (codegen without
  compiling) checked for all structural pieces.
- t_size_not_in_signature_regression: the exact bug scenario.

CI: the codegen-compile-check job now also compiles CircularTensor sources
(packed/planar, newest/oldest, with preproc chains) via generate_source().
…n tests

Closes out the pending-work checklist:

- CPU backend (target='cpu'): generates a plain C++ TU running
  TransformDPP<ParArch::CPU> on MemType::Host pointers, compiled by
  clang++ (g++ rejects Stream_<ParArch::CPU>'s ctor spelling), numpy
  arrays in/out. 5 tests incl. CPU==GPU cross-validation.

- ThreadFusion (thread_fusion=True): emits TransformDPP<GPU_NVIDIA,
  TF::ENABLED>. Auto-falls back to scalar for shapes whose row bytes are
  not 16-aligned: external tight-pitch pointers fault on vectorized loads
  (found via real 'misaligned address' on a 251-wide image). 5 tests:
  exactness vs scalar baseline, odd widths, uchar3, distinct cache keys.

- Multi-GPU: DeviceBuffer/CircularTensor take device=N (per-device primary
  contexts retained via the driver API); compose pipelines follow the
  input's device (torch cuda:N / cupy / DeviceBuffer); DLPack device id
  propagated. Honest caveat: single-GPU box, exercised on device 0 with
  clean errors for absent devices.

- Torch integration suite (torch 2.12+cu130, real GPU): zero-copy both
  directions (mutation-visibility proven), external torch streams, out=,
  uchar3 HWC->CHW vs torch reference, CircularTensor->torch window, batch
  HF from torch list. Found+fixed _alloc_out shape bug for TensorSplit
  chains (was returning HWC-shaped tensor for planar CHW output).

- fk::Equal / Tuple-input ops: documented as structural limit (needs a
  multi-source Read producing Tuple<A,B>; upstream feature candidate).

- spikes/repro_divergent_sum_z.cu: reproducer behind upstream issue #250
  (Divergent Executor launches sum-of-z planes; compute-sanitizer verified
  OOB + canary corruption).

Full matrix: 13 suites x 2 backends (nvcc+clang) + cpu + torch, all green.
@morousg
morousg merged commit 2db8b41 into Libraries-Openly-Fused:main Jun 11, 2026
4 checks passed
morousg pushed a commit that referenced this pull request Jun 24, 2026
…yone

DivergentHF: fix off-by-one in sequence selector (0-based kernel)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants